Automatic tracking matte system

ABSTRACT

A system for generating automatically tracking mattes that rapidly integrates live action and virtual composite images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending InternationalApplication No. PCT/US12/67460 filed Nov. 30, 2012 and which claims thebenefit of provisional application Ser. No. 61/565,884, filed Dec. 1,2011, and the entire contents of both of these prior applications arehereby incorporated by reference.

TECHNICAL FIELD

This invention relates to systems and methods for combining real sceneelements from a video, film, digital type camera or the like withvirtual scene elements from a virtual camera into a finished compositeimage, and more particularly, to systems and methods for creating drawnshapes that move along with the camera's motion to control how thevirtual and real scene elements are combined.

BACKGROUND

The state of the art in combining real world imagery with additionalimagery from another source is a process that requires careful controlover which sections of each image are to be used in the final compositeimage. One common application is to combine images generated by acomputer with images acquired from a traditional motion picture, videoor digital camera. In order to seamlessly combine the images, the areasof each image that are to be preserved or modified must be defined.These areas are typically called mattes.

Mattes may be defined in a number of ways. In traditional compositing,the mattes are frequently defined by having an artist mark points aroundthe perimeter of the object to be preserved or removed. The computerthen connects the dots to form a closed shape, which forms the matte.Problems can arise, however, if the object and/or the camera moverelative to the other.

In traditional computer compositing, a moving camera or object ishandled by making a tracking matte, or a matte that moves along with theobject. While the methods of moving the matte along with the objectvary, they typically center around having the user specify an area ofhigh contrast in the live action image, measuring how that image movesaround in the frame, and connecting the motion of the drawn matte to themotion of the high contrast object.

This process works, but has several limitations. Firstly, if the highcontrast area is located on the front of a character's shirt, forexample, and the character turns around, or if the camera moves aroundto another side of the character, the local effect is destroyed.Secondly, the process of measuring the camera motion by tracking theindividual pixels of the high contrast part of the image is both fragileand time-consuming if there is no additional camera data to work from.It typically cannot be computed in real time, and if a frame of the liveaction image has a lighting change where the pattern is unrecognizable,the artist must re-specify the high contrast area at the frame offailure to continue the process. The process of creating all of themultiple overlapping mattes that are used in a sophisticated visualeffects shot can exceed the time required to complete the rest of theshot due to the handwork required.

In addition, if the live action camera is zoomed in, the high contrastarea that was being tracked can simply disappear from the image,resulting in the matte failing to track the camera lens change.

Accordingly, the pixel tracking based methods do not work well for thedemands of real time visual effects processing, which must be very rapidto compute as well as robust to the frame by frame changes in the liveaction video image.

In real time processing, mattes have traditionally been created bysurveying the edges of the green screen background using anarchitectural measurement tool such as a total station, and creating amodel of the matte in 3D space. However, models of this type cannot berapidly modified by the artist under typical time pressure conditionsfound in entertainment production.

SUMMARY

Various embodiments of an automatic tracking matte system are disclosedherein. In one embodiment, an artist selects points on a computer screento generate a rough outline around the object to be removed orpreserved. These points are selected using a 2D display of the liveaction image, typically by locating a pointer in the desired locationand pressing a selection button. The user clicks a mouse around theborder of the object, and then selects the inside or the outside of thefinished outline to determine on which side of the line the matte willbe active. The user can also begin or end the outline at an edge of thescreen, in which case the system extrapolates the matte for a givendistance out from the edge of the screen. This distance can be fivemeters or more, generally between one and ten meters.

The above process generates a 2D outline. However, for the matte totrack properly in a 3D space, the shape must be converted to a 3Drepresentation. This 3D shape can be a set of attached polygons whoseouter perimeter matches the outline of the points that the userselected. The 3D polygon mesh exists at a given point in 3D space. The3D polygon mesh can be created in a plane normal to the axis of the mainvirtual camera when the matte is initiated, and at a distance specifiedby the user.

Since the mesh is created by drawing around a live action object, the 2Drepresentation is viewed from the position of the current live actioncamera. For the 3D mesh to line up accurately, it can be projected froma virtual camera with the same position and orientation as the liveaction camera. In addition, the further away from the camera the polygonmesh is moved, the larger it must become for the 2D points to remain inthe same relative position on the live action image. This computationcan be done automatically by geometric projection as the user moves the3D polygon mesh closer or further away from the virtual camera. Thisautomatic calculation can take into account the current position andorientation of the virtual and live action cameras, the current focallength and distortion of the cameras, and the sensor size of thecameras.

After creating the mesh, the user will frequently need to adjust theposition and/or shape of an existing mesh. The camera may have moved inthis interval, but to keep the points aligned correctly with theoriginal object, the normal along which the 3D mesh is scaled up or downmust be known. The mesh points can be manipulated by the artist directlyin the 2D user interface, but may be constrained to move only in theoriginal 3D plane in which they were created.

According to an aspect of the disclosure a unified matte system iscreated with individual points that are entered either on the screen ina 2D form as described herein, or directly in 3D from survey coordinatedata. Once a given polygon is entered, the various points can be forcedinto a plane. This plane then defines where the individual points canmove when later edited. Thereby, the artist can simply click and drag onan existing matte point to edit it, knowing that it will stay in theplane in which it was created.

According to one aspect of the disclosure the 3D mesh object(s) is (are)rendered in separate passes, and grouped together to form the overallset of despill, garbage, or other types of mattes.

According to another aspect of the disclosure the points of the mesh canalso be entered using 3D survey data. This 3D data can be determined ina variety of ways, including photogrammetry techniques and lasersurveying instruments such as a total station. In this embodiment, thefirst three entered points of survey data can be used to set the planeof the rest of the entered survey points of that polygon.

According to a further aspect of the disclosure the mesh can be made tomove along with a separate form of tracking. For example, a separatemotion capture system can measure the 3D location of a person, face, orobject in real time, and locate the 3D matte mesh at the location of theperson.

According to a still further aspect of the disclosure the basic matteshape can be used for many different applications such as a garbagematte (removal of foreground), a despill matte (removal of extra blue orgreen color), a color grading matte (selective enhancement of one areaof the scene's color), and so forth.

According to a yet still further aspect of the disclosure the mattedistance set can be set automatically by measuring the distance from thecamera to the subject, such as by acoustic or optical methods, or bymeasuring the current focus distance from the lens system.

According to an aspect of the disclosure a method for creating matteswhose shape can be drawn by an artist, but which tracks automatically asthe camera or object moves, is provided.

According to another aspect of the disclosure the computations requiredto move the mattes can be performed in real time.

According to a further aspect of the disclosure the matte tracking canautomatically handle variations in lens focal length or distortion.

According to a still further aspect of the disclosure the matte data canbe entered in standard 3D survey coordinate form and rapidly modified bythe artist during production.

According to another aspect a matte tracking method can be achieved withdata that is already existing in a real-time compositing and cameratracking system.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the present inventionwill be more fully understood from the following detailed description ofillustrative embodiments, taken in conjunction with the accompanyingdrawings.

FIG. 1 is a perspective view of an embodiment in accordance with thepresent disclosure.

FIG. 2 is a front view of a live action image with a set of pointsaround it representing a rough matte outline in accordance with anembodiment of the present disclosure.

FIG. 3 is a perspective view of a 3D compositing scene in accordancewith the present disclosure.

FIG. 4 is a perspective view of a polygon mesh in a 3D compositing scenein accordance with the present disclosure.

FIG. 5 is a top view of a 3D compositing scene in accordance with thepresent disclosure.

FIG. 6 is a perspective view of a live action environment located withina coordinate system in accordance with the present disclosure.

FIG. 7 depicts a matte outline before and after applying lensdistortion, which uses the present disclosure to generate anautomatically tracking matte.

FIG. 8 is a block diagram that depicts the data flow through a system ofthe present disclosure.

DETAILED DESCRIPTION

The following is a detailed description of the presently known bestmode(s) of carrying out the inventions. This description is not to betaken in a limiting sense, but is made merely for the purpose ofillustrating the general principles of the inventions.

A rapid, efficient, reliable system is disclosed herein for generatingan automatically tracking matte that significantly speeds theintegration of live action and virtual composite images. Applicationsranging from video games to feature films can implement the system in afraction of the time typically spent tracking multiple areas of highcontrast in the image by hand. The system thereby can greatly reduce thecost and complexity of controlling matte motion, and enables a muchwider usage of the virtual production method.

Since the present process is primarily for joining live action withcomputer-generated elements, its applications for video games may belimited. The process can work with a real-time video feed from a camera,which is presently available on most “still” cameras as well. Theprocess can work with a “video tap” mounted on a film camera, in systemswhere the image is converted to a standard video format that can beprocessed.

An objective of the present disclosure is to provide a method andapparatus for creating automatically tracking mattes for a live actionsubject that enable rapid control over how different areas of the imageare processed.

Referring to FIG. 1, an embodiment of the present disclosure isdepicted. A scene camera 130 with a lens 190 is positioned to capture alive action image 20 of a subject 10 in front of a background 100. Thesubject(s) 10, for example, can be actors, props, and physical sets. Thebackground 100 may have obstructions 115 that may cause problems withthe keying process. Per an aspect of the disclosure, the obstructions115 can be dealt with by creating matte objects of this disclosure.

The scene camera 130 can be mounted on a camera tracking system 140. Andthis camera tracking system 140 can be an encoded pedestal, dolly, jib,crane, or any other form of camera position, orientation, andfield-of-view measuring system. Focus distance may also be measured, asthe parameters of a lens can change while focusing. There may be morethan one scene camera to enable different views of the subject'sperformance to be captured.

The scene camera 130 and the camera tracking system 140 are connected toa video processing system 150, as depicted in FIG. 8. The videoprocessing system 150 takes the incoming live action video image 20,generates the corresponding background virtual matte shape 60, andperforms the automatic tracking matte process using the two images. Thevideo processing system can include a computer with a live video input,a camera tracking data input, and a video card capable of processing 2Dand 3D computer graphics calculations.

An embodiment of the present disclosure is illustrated in FIG. 2. A liveaction subject 10 is shown in the center of a live action image 20 witha perimeter 30 around the edge. A matte shape 60 is displayed along withthe live action image 20 in a user interface 240. The matte shape 60,outlined by a set of points 40 connected by segments 50, is drawn aroundsubject 10. Perimeter points 42 are similar to points 40, but arelocated within the boundary set by perimeter 30. The user selects thelocation of points 40 one at a time by clicking on the screen.

Pursuant to one embodiment, if the user starts and ends the shape bycreating a perimeter boundary point 42 within the perimeter 30, thematte shape 60 will extend off the screen in the direction of the lines50 as they extend off the screen. On the other hand, if the user selectsthe start point after selecting several other points, the program willrecognize this as a closed ring.

Once the matte outline 50 has been defined, the user selects whether thematte shape 60 is to be on the inside or the outside of the outline 50.This can be done by detecting to which side of the closed shape that theuser moves the mouse pointer, and then clicks to set the inside or theoutside of outline 50 to select.

The display in the user interface is 2D, but for correct alignment, allof the various components exist as 3D objects in a virtual scene.Referring to FIG. 3, the transformation of 3D to 2D coordinates can bedone using standard projection geometry calculations that are well knownto those skilled in the art. A virtual camera 70 has a frustum 80 thatdescribes the field of view of the virtual camera. The parameters ofvirtual camera 70 and frustum 80 match the parameters of the live actioncamera 130 and the lens 190.

A live action image 20 containing a subject 10 is located at a distancefrom virtual camera 70, and is centered on the optical axis 82 of thevirtual camera. The size of the live action image 20 in the virtualspace is determined by its distance from virtual camera 70; the furtheraway the live action image is placed in the virtual space, the largerthe image must be to fill the view angle described by virtual frustum80. The matte shape 60 is shown as located in the 3D space in betweenvirtual camera 70 and live action image 20. Since the user controls thedistance between virtual camera 70 and matte shape 60, the matte shapecan also be located further away from the virtual camera than liveaction image 20. In this image (see FIG. 3), an automatic extension 62of matte shape 60 is seen; this extrapolates the direction of segments50 that end in perimeter points 42 to go past the limits of the screen.The automatic extension 62 can extend five meters, for example, past theoriginal visible edge of matte shape 60.

The orientation of matte shape 60 can be created perpendicular to theoptical axis 82, and at a user specified distance from the virtualcamera 70. Thereby, the user can measure out how far away the liveaction subject 10 is, enter that distance into the interface, and knowthat the matte shape 60 is being created at a matching distance from thevirtual camera 70.

When entered as 2D points on a plane normal to the user's viewing axis,the points 40 all lie on the same plane. Matte shape 60 can also becreated using direct input of 3D survey data, measured with anarchitectural survey tool such as a total station. The entered points ofmatte shape 60 can be forced to lie on the same plane by using the firstthree entered points to set the plane definition, with additionalentered points projected into that plane to enforce planarity.

To correctly render a 3D shape, the outline can be broken up intoindividual triangular elements. FIG. 4 demonstrates an embodiment ofthis method. The outline 50 is automatically converted into a set ofpolygons 90 with internal edges 92 using an automatic tessellationroutine. The automatic extension 62 is similarly tessellated. Thisautomatic tessellation routine can be done using an algorithm calledDelaunay triangulation as is well known to practitioners.

As the user can adjust the distance of the matte shape 60 from thecamera, the size of matte shape 60 must increase and decrease as it ismoved closer to or further away from the virtual camera 70. FIG. 5demonstrates an embodiment of this process wherein virtual camera 70 andvirtual frustum 80 are viewed from the top down to make their geometryclearer. Three positions of matte shape 60 are shown increasing anddecreasing in size as they are closer or further away from virtualcamera 70. The size of matte shape 60 is scaled in proportion to theviewing angle or field of view of virtual frustum 80.

The user can also adjust the overall matte shape 60 by moving the points40 after the original shape has been created. The points 40 can beconstrained to their original created plane in 3D space as they aremoved around. This enables the artist to manipulate points using aconvenient interactive 2D interface common in computers, but have thepoints stay in the correct 3D plane.

In some cases the matte shape 60 will need to move along with thesubject 10. This can occur when a foreground subject 10 is moving. (Onthe other hand, the matte shape does not need to move when it is drawnaround a background object, such as a green screen wall, that does notmove.)

Referring to FIG. 6, a 3D tracking device 130 can be used to measure theposition 120 of the subject 10 in the stage. The position 120 of subject10 is measured with respect to a coordinate system 110. This coordinatesystem 110 can be located identically to the virtual coordinate systemused for the rest of the background. The 3D tracking device 130 can beany type of system that can resolve the location of the subject 10 onthe stage; and as an example it can be a markerless motion capturesystem. Since the position 120 of the subject 10 is known by the system,the position of matte shape 60 can be connected to the position 120 ofsubject 10, with the result being that the movement of matte shape 60will be locked to subject 10 even as both subject 10 and virtual camera110 move around the scene. The orientation of matte shape 60 can changeto remain normal to the virtual camera 70 as it is moved.

All physical lenses exhibit distortion, which must be handled tocorrectly match the matte shapes to live action, an example of which isshown in FIG. 7. As before, matte shape 60 is created by connectingsegments 50 together and the end points of segments 50 are the selectedpoints 40. However, the live action image from which the user isselecting points 40 has lens distortion. To create a correct matteobject 60 that works correctly in 3D coordinate space, the selectedpoints 40 in the interface have distortion, which is removed beforebeing converted to a polygon mesh.

To render a matte shape 60 that correctly fits to distorted points 40,an undistorted matte shape 62 is created by generating undistortedpoints 44 based on applying lens distortion removal calculations to theoriginal segment points 40, and connecting them with undistortedsegments 52. The calculation of undistorted points 42 on the X,Y planeof the user interface and the rendered matte from the original pointslocation can be computed with the following equations:Xundistorted=Xdistorted*(1+K1*radius²)Yundistorted=Ydistorted*(1+K1*radius²)

The value of K1 can be generated by a lens calibration system thatmeasures the current distortion of the physical lens at its currentsetting. An example of a lens calibration system is described in U.S.patent application Ser. No. 12/832,480, which was published as U.S.Patent Publication No. 20110026014 and whose entire contents are herebyincorporated by reference. The conversion of the undistorted points 42and segments 52 into 3D coordinates can be completed with standardprojection geometry calculations well known to practitioners in thefield. To then display the correctly distorted matte shape 60, theundistorted matte shape 62 can be rendered in 2D space and the reverseof the above distortion calculations can be applied to it. In this way,the undistorted matte shape 62 is properly displayed no matter what thecurrent live action lens is doing.

The data flow of the system is illustrated in FIG. 8. A number of theprocessing steps described in earlier figures are combined into thevideo processing system 150. A scene camera 130 transmits a live actionimage 20 to 2D compositor 180. Camera tracking system 140 measures andtransmits camera data 160 to data combiner 300. Lens 190 transmits lensposition data 200 to the lens calibration table 210. Lens calibrationtable 210 looks up the appropriate lens data 230 and transmits that datato data combiner 300. Data combiner 300 then transmits the combinedcamera and lens data 310 to the 3D renderer 290, to distortion removalprocessor 260, to 2D-to-3D converter 270, and to distortion additionprocessor 170.

The user clicks perimeter points 40 and perimeter boundary points 42 onthe user interface 240, which transmits these points to the distortionremoval processor 260. Using the combined data 310, which includes lensdata 230, the distortion removal processor 260 creates a set ofundistorted points 44 that are transmitted to the 2D-to-3D converter270. The distortion removal processor 260 can use the distortionalgorithms mentioned with respect to FIG. 7. Using the current cameraand lens data contained in combined data 310, the 2D-to-3D converter 270calculates the matte shape 60 and sends it to 3D renderer 290. Thecalculation of the 3D matte geometry based on the undistorted points 44and the combined data 310 can use projection geometry calculations thatare well known to those skilled in the art.

3D renderer 290 can use matte shape 60 and the combined camera and lensdata 310 to place a virtual camera 70 and frustum 80. The 3D renderergenerates a 2D undistorted matte shape 62. The creation of a 2Dundistorted shape from 3D geometry is essentially the reverse of the2D-to-3D conversion mentioned in the previous paragraph, and is wellknown to those skilled in the art. The 3D renderer 290 then sends the 2Dundistorted matte shape 62 to the distortion addition processor 170. Thedistortion addition processor 170, using the lens data 230 contained incombined data 310, creates a distorted 2D matte image 175 and sends itto 2D compositor 180. The calculations to add this distortion can be thesame as described for FIG. 7.

A goal of this 2D-to-3D and 3D-to-2D conversion is to allow the user toselect and manipulate points on a 2D user interface 240 containing liveaction image 20 that actually generate correct matte shape 60 which whenrendered with the same lens distortion as the live action image 20,results in a matte image 175 that lines up with the original perimeterpoint 40 selected by the user. Otherwise, the matte image 175 wouldappear in a different place than that selected, and this would be afrustrating interface for the user.

The same rendering and distortion addition process can be used to createvirtual background scenes that will be combined with the live actionimage 20 in the 2D compositor 180. Background scene geometry 320 from anexternal 3D content creation software program such as Maya is loadedinto the 3D renderer 290, which generates an undistorted backgroundimage 340. This is sent to the distortion addition processor 170, whichthen applies the same lens distortion addition used for the matte image175 to result in background image 185.

2D compositor 180 uses the matte image 175 to selectively processportions of the live action image 20 in combination with backgroundimage 185 to generate a composited image 320. (The composited image 320can be delivered in the form of a live action actor placed into avirtual background, for example.) Because of the correct removal,rendering, and addition of lens distortion information, the user simplyclicks on perimeter points 40 and they appear correctly on the screen ofthe user interface 240 in the expected position. This is because theyhave been correctly converted to accurate 3D spatial coordinates andre-drawn with matching lens and camera data. Thus, the convenience of 2Ddrawn mattes is preserved, while operating in a fully-tracked 3D world,which is needed for complex real-time visual effects.

According to one program of a system of this disclosure, the followingprompts are provided to the user at the user interface: selectable anddraggable points that overlay a live action image. An alternativeprogram provides the following prompts: numerical XYZ entry fields fordirect input of 3D coordinate points.

The resulting drawn or surveyed mattes can be used in a variety ofmanners. The mattes can be used as a garbage matte or a despill matte.

Garbage mattes are used to completely remove unwanted sections (like ahanging microphone in front of the green screen) of the live actionimage. The garbage mattes replace that part of the live action imagewith the computer-generated image underneath.

On the other hand, despill mattes are used to preserve part of theforeground image from being keyed (the green area made transparent), butstill “clamping” the green (limiting the green level to the largest ofeither the red or the blue levels) to remove the greenish cast thatotherwise permeates all through the image from the reflected light offthe green screen. An example is a green screen placed outside a window,but the green reflects onto a glass table indoors, making it green. Adespill matte removes the green tinge from the glass top, but withoutmaking it transparent. That is, a despill matte defines the part of thelive action foreground to apply only the despill process, as opposed tothe keying process, both of which are well known to practitioners in theart.

An alternative embodiment is the creation of the ‘holdout’ matte. Thisis typically based on live action objects in the scene, and is used toforce virtual objects to be behind the live action objects, or to enablevirtual objects to cast virtual shadows on live action objects. This isthe area of use most likely for 3D mattes generated from natural featuretracking.

In addition, the 3D objects that are used to describe the mattepositions can be saved and exported to external applications forpost-production. They can be saved into a Collada or other 3D fileformat that is easily imported into other standard visual effectsapplications.

Alternative embodiments include using the mattes to drive a colorgrading process, so that the matte defines the part of the image towhich to apply a color transformation. In this way, the process ofcorrecting images manually, shot by shot, can be heavily automated.

Additional alternative embodiments include the automated movement ofdifferent points in the matte according to different tracking pointsfrom a 3D tracking system, or using facial tracking connected to themain camera to drive the matte tracking to only track facial features.

Thus, systems of the present disclosure have many unique advantages suchas those discussed immediately below. The artist can edit the 3D pointsby dragging them around in a 2D interface, while preserving theirlocation on their original 2D plane. This gives the artist fastinteraction, while avoiding confusing “out of plane” geometry. Using a2D interface can be accomplished by real time undistortion andre-distortion, to create correctly matched geometry while providing aconvenient, familiar 2D interface. Most compositors only work with 2D,and 3D can be confusing to them. Automatically extending the mattebeyond the edges when using the perimeter points allows the compositorto extend the matte without requiring the camera operator to move backand forth. The system allows the mattes to be stored and exported forfuture use, which is particularly useful for example for the followingapplications: Nuke, After Effects, Shake, Flame, and Inferno.

A system of the present disclosure can include a graphics card or CPUthat includes: (a) a distortion removal processor 260 programmed tocreate a set of undistorted points; (b) a 2D-to-3D converter 270configured to use the set of undistorted points to calculate 3D mattegeometry; (c) a 3D renderer 290 configured to use the 3D matte geometryto generate a 2D undistorted matte shape; (d) a distortion additionprocessor 170 programmed to use the 2D undistorted matte shape to createa distorted 2D matte shape; and (e) a 2D compositor 180 configured touse the distorted 2D matte shape to combine at least one portion of alive action image with at least one other image to generate a compositedimage. The composited image can be delivered in the form of a highdefinition serial digital interface signal to an external recordingsystem. An example of a commercially available graphics card that can beso programmed is the Quadro card available from nVidia Corporation ofSanta Clara, Calif.

The above-mentioned graphics card or CPU can also include data combiner300 and lens calibration table 210, or the processes can be dividedbetween a graphics card and a CPU.

The present automatic matte tracking system can be based on the priorart Previzion system, which is/was available from Lightcraft Technologyof Venice, Calif. The Previzion system includes a camera trackingsystem, a lens calibration system, a real-time compositing system, and abuilt-in 3D renderer. The tracking mattes feature adds the ability tohand draw mattes in 2D on the screen, that are then converted into a 3Dspace by the system, enabling it to move automatically as the cameramoves, and in real time. An example of a publication disclosing theprior art Previzion system is the Previzion product brochure, entitledPrevizion Specifications 2011, published on Apr. 8, 2011, and whosecontents are incorporated by reference.

An embodiment of the present system can be made by modifying the priorart Previzion system by adding a tab to the user interface where theuser can create the present matte and adjust it. The prior art Previzionsystem can be adapted by the addition of the drawable mattes, thecomputations of their positions and orientations and their adjustmentsusing the saved common plane of the 3D points.

Previzion is unique in that the 2D video processing and the 3D renderingare being done in the same product. In contrast, most other systems haveseparate consoles for 2D and 3D, which are used to separately create the3D background virtual scene and merge it with the 2D live action scene.

However, the 3D box that has the tracking matte information can send itto the 2D box, in the form of another 2D video signal that is ablack-and-white garbage matte. This would essentially be the 3D boxrendering the matte shapes, as it does in the Previzion system, but thefinal image assembly would be done externally in another system (like anUltimatte HD, which is available from the Ultimatte Corporation ofChatsworth, Calif.) that takes in both the black/white garbage mattesignal and the live action blue or green screen signal.

Most Ultimatte/other third party keyers already have a live input forthe garbage matte signal, so it is straightforward to interface thetracking garbage mattes of the present disclosure to external keyers.However, the 2D Ultimatte system has no user interface that can selectpoints that are connected to the separate 3D rendering system, such asis described here.

The more complicated uses of the mattes (like despill, color correction,etc.) that are easy to do in Previzion can be re-created with anexternal keying system. They can be done, for example, by manuallytracking points of high contrast in the 2D image in Nuke available fromThe Foundry Visionmongers Ltd. of London, UK, or similar compositingpackages, and then creating outlines from these points. This istypically not a real time process, and requires days or weeks of workfor a single shot.

Pursuant to an aspect of the present disclosure what the camera andcamera lens are doing are knowable to the present system. Thus, 2D-3Dand 3D-2D conversions can be done quickly while taking into account lensdistortion. The distortion removal processor, 2D-to-3D converter, 3Drenderer, distortion addition processor, and 2D compositor can all beperformed on a graphics card of the system. A video I/O card handles thevideo input and output.

A program of the present disclosure can be delivered as an executablecode that is installed on a target system. The same math can work in abrowser as it is largely a matter of geometry and input.

Although the inventions disclosed herein have been described in terms ofthe preferred embodiments above, numerous modifications and/or additionsto the above-described preferred embodiments would be readily apparentto one skilled in the art. The embodiments can be defined, for example,as methods carried out by any one, any subset of or all of thecomponents as a system of one or more components in a certain structuraland/or functional relationship; as methods of making, installing andassembling; as methods of using; methods of commercializing; as methodsof making and using the terminals; as kits of the different components;as an entire assembled workable system; and/or as sub-assemblies orsub-methods. It is intended that the scope of the present inventionsextend to all such modifications and/or additions and that the scope ofthe present inventions is limited solely by the claims set forth below.

What is claimed is:
 1. A system comprising: a video processor configuredto convert a matte shape to 3D spatial coordinates and redrawn with lensand camera data matching the live action camera and lens and to use thematte shape to selectively process portions of a live action image incombination with at least one other image to generate a compositedimage; wherein the matte shape is defined by matte points lying in aplane normal to the line of sight of the live action camera.
 2. Thesystem of claim 1 wherein the live action camera has an adjustable lens.3. The system of claim 2 wherein the matte points are manually-selectedmatte points.
 4. The system of claim 1 wherein the matte shape isdefined at least in part by a recording of the original camera location,focal length and distance between the camera and the plane of the matteat the time of matte creation.
 5. The system of claim 1 wherein thematte shape is defined by a scaling of the matte shape proportionally tothe original focal length of the camera and along the original cameraaxis as the user adjusts the matte distance after the matte has beencreated.
 6. The system of claim 1 wherein the matte shape is defined bya translating the scene camera by the user, and then an adjusting of thedistance of the matte plane from the original camera location until thematte points line up with their respective scene elements in the newcamera position, thereby resolving the position of the matte shape withthe object being outlined in 3D space.
 7. The system of claim 1 whereinthe matte shape is defined by a constraining of the drawn points so thatthe points can be moved closer or further away from the original cameralocation, but only along the optical projection axis of the originalcamera and constrained to a plane normal to that of the original cameraaxis at time of the matte was originally created.
 8. The system of claim1 wherein the user sets the matte distance by directly measuring thedistance between the live action camera and the live action object thatis to be matted at the time of matte creation.
 9. The system of claim 1wherein the at least one other image is a virtual background createdwith the same camera perspective as the live action foreground.
 10. Amethod comprising: selecting points on a computer screen to generate arough outline around an object to be removed or preserved from a liveaction image from a live action camera; the selecting includingbeginning or ending the outline at an edge of the screen; using acomputer, converting the outline to a matte that lies in a planeperpendicular to an optical axis of the live action camera; thebeginning or ending causing the matte to be automatically extended apredetermined distance out from the edge of the screen without requiringthe camera to move back and forth; and the matte being comprised ofthree-dimensional geometry in an arbitrary three-dimensional plane, sothat areas extended past the geometry visible to the user are projectedinto three-dimensional space.
 11. The method of claim 10 wherein thelive action camera has an adjustable lens.
 12. The method of claim 11wherein the plane can be adjusted further or closer to the original liveaction camera location at time of matte creation.
 13. The method ofclaim 1 wherein the selected points are scaled along the originalcamera's axis and focal length projection as the user changes the matteplane distance.
 14. The method of claim 1 wherein the selected pointsare viewed from the new camera position with correct distortion of thelens in the new position.
 15. The method of claim 11 wherein the mattecan be scaled closer or nearer to the original camera location evenafter the scene camera has been moved, so that the user can triangulatethe 3D position of the matte in space by aligning the matte to liveaction features in two camera locations.
 16. The method of claim 11wherein the selecting includes restricting the 2D dragging movement ofthe matte points within a plane normal to the original camera axis. 17.A method comprising: manually generating a 2D outline on a computerscreen around an object to be removed or preserved in a 2D display of alive action image from a live action camera; using a computer,converting the 2D outline to a 3D mesh on a plane normal to the opticalaxis of the original live action camera location; and projecting the 3Dmesh from a virtual camera, which is defined in the computer, having thesame position and orientation as the live action camera to line up the3D mesh with elements of the live action image.
 18. The method of claim17 further comprising the 3D mesh at least approximately tracking thelive action object as the live action camera is moved to a new positionrelative to the object, and the 3D mesh scaled relative to the originalcamera position, orientation and focal length until the 3D mesh alignswith the live action object from the new position, orientation and focallength of the live action camera.
 19. The method of claim 18 wherein thetracking includes adjusting the location of the points of the outline inthree dimensional space so that the 3D mesh scales as the distance fromthe original camera location changes.
 20. The method of claim 17 whereinthe 2D outline lies in a plane perpendicular to the line of sight of thelive action camera.
 21. The method of claim 17 wherein the generatingincludes manually adjusting the distance between the original cameraposition and the normal plane on which the selected matte mesh pointsare located.
 22. The method of claim 17 wherein the projecting includesmoving the live action camera to a new location, and adjusting thedistance between the matte plane and the original camera until the mattepoints line up with the live action object in the new live action cameraposition and orientation, thereby locating the matte object in 3D space.23. The method of claim 17 further comprising after the projecting the3D mesh, manipulating the points in a 2D interface while constrainingthe points to move on a plane normal to the camera axis in which thepoints were originally created.
 24. A method comprising: manipulating a3D object, which is defined in a computer, via a 2D interface by forcing3D points to lie in a plane and calculating their 3D positions bydragging around a 2D dot on a 2D interface; the plane beingperpendicular to the tracked camera position from where the points wereoriginal drawn; and setting the distance between the matte plane and thescene camera location by measuring the distance between the live actioncamera and the live action object using a measuring device.
 25. Themethod of claim 24 wherein the measuring device is a ruler or anelectronic rangefinder.
 26. The method of claim 24 wherein themanipulating includes tracking at least approximately the 3D object bylocating the 3D matte at approximately the same position as the actuallive action object via direct measurement of the distance between thecamera and the 3D object at time of the matte creation, or triangulatingthe distance to the 3D object by translating the scene camera andadjusting the distance between the matte object and the original camerauntil the matte object lines up with the live action object in the newcamera position.
 27. The method of claim 24 wherein the live actioncamera has an adjustable lens.
 28. A method comprising: generating a 2Doutline around an object to be removed or preserved in a 2D display of acomputer of a live action image from a live action camera; using thecomputer, converting the 2D outline to a 3D mesh on a plane normal tothe optical axis of the original live action camera location; projectingthe 3D mesh from a virtual camera, which is defined in the computer,having the same position and orientation as the live action camera toline up the 3D mesh with elements of the live action image; and afterthe projecting the 3D mesh, manipulating the points in a 2D interfacewhile constraining the points to move on a plane normal to the cameraaxis in which the points were originally created.
 29. The method ofclaim 28 further comprising the 3D mesh at least approximately trackingthe live action object as the live action camera is moved to a newposition relative to the object, and the 3D mesh scaled relative to theoriginal camera position, orientation and focal length until the 3D meshaligns with the live action object from the new position, orientationand focal length of the live action camera.
 30. The method of claim 29wherein the tracking includes adjusting the location of the points ofthe outline in three dimensional space so that the 3D mesh scales as thedistance from the original camera location changes.
 31. The method ofclaim 28 wherein the generating includes manually adjusting the distancebetween the original camera position and the normal plane on which theselected matte mesh points are located.
 32. The method of claim 28wherein the projecting includes moving the live action camera to a newlocation, and adjusting the distance between the matte plane and theoriginal camera until the matte points line up with the live actionobject in the new live action camera position and orientation, therebylocating the matte object in 3D space.
 33. A method comprising:manipulating a 3D object, which is defined in a computer, via a 2Dinterface on the computer by forcing the object's 3D points to lie in aplane and calculating their 3D positions by dragging around a 2D dot ona 2D interface; the plane being perpendicular to the tracked cameraposition from where the points were original drawn; and in themanipulating including using the computer to track at leastapproximately the 3D object by locating the 3D matte at approximatelythe same position as the actual live action object via directmeasurement of the distance between the camera and the 3D object at timeof the matte creation, or using the computer to triangulate the distanceto the 3D object by translating the scene camera and adjusting thedistance between the matte object and the original camera until thematte object lines up with the live action object in the new cameraposition.
 34. The method of claim 33 wherein the live action camera hasan adjustable lens.